Factors Impacting Novice Code Comprehension in a Tutor for Introductory Computer Science

To test the effect of questions regarding the low-level details of a program compared to questions regarding the more abstract nature of the algorithms, a tutoring environment was constructed. Prior work has shown a strong correlation between students’ perception of the global goals of a program in a code comprehension task and their ability to write code to solve problems[1; 2; 3]. Using these tutoring data, we model the students’ learning progression through the activities and compare student performance on common tutoring questions. The models were evaluated for predictive factors in student performance. Performance on common questions was dependent upon the factors of the design as well as the students’ performance and how many questions they had completed. These results can help give us insight into the type of practice students need to better comprehend code and points to types of feedback to explore in future work. In the fall of 2010, introductory computer science students from Carnegie Mellon University (N=236) participated in a study to determine the effects of interacting with a code comprehension exercise. Students engaged with the tutor as a part of a homework assignment in their introductory computer science course. Three different instructors teach the course, and students were drawn from all classes. Students were randomly assigned to conditions pairing Abstraction vs Tracing with Context vs. NoContext. In the contextualized conditions, the examples that students saw included a one-sentence description of the context, and variables names were changed to match the context. As a part of the tutor, students interacted with 30 code comprehension questions. The 30 questions referred to about 10 different code segments, implementing various algorithms (e.g., sum, search, sort, rotate). Each code segment had a series of three questions associated with it. The first question was a tracing question and was seen by all students, regardless of condition, and is referred to as the common question. The second and third question varied by condition. In the Tracing condition, students were asked to trace the code with two other data sets. In the Abstract condition, students were asked questions about the global goals of the algorithm, or the role of its components. Code examples were presented to all students in the same order, and the first 6 were selected to be easier than the last 4.